[00:00.000 --> 00:07.000]  Well, thank you, Ajit. I want to welcome everybody to DEF CON and the Blockchain Village.
[00:07.000 --> 00:11.300]  My name is Ron Stoner. I'm going to talk about securing the cosmos, and I appreciate that
[00:11.300 --> 00:19.600]  wonderful introduction. I'm a repeat speaker here. I spoke last year when we were talking about the
[00:19.600 --> 00:24.200]  cryptocurrency security standard with Michael Parkland, and it was an amazing experience,
[00:24.200 --> 00:29.680]  so I'm happy to be back. I will preface this talk by saying I'm only on a few hours of sleep
[00:29.680 --> 00:33.860]  in traditional DEF CON fashion, a little bit hungover, and I was working on my presentation
[00:33.860 --> 00:38.700]  up until a few minutes ago. So if there's any inconsistencies in here, or you guys see
[00:38.700 --> 00:43.040]  formatting issues, I'm more than happy to clean that up if you want to copy the presentation,
[00:43.040 --> 00:47.660]  or if you want to discuss further just to clear up any ambiguity after the stream.
[00:49.000 --> 00:56.080]  So who am I? I'm a senior security engineer for ShapeShift.com, one of the best digital
[00:56.080 --> 01:00.840]  asset exchanges out there. And as a result, I get exposed to a lot of different technologies,
[01:00.840 --> 01:04.280]  and I get to play with a lot of cool different things in the crypto space,
[01:04.280 --> 01:10.960]  including Cosmos, the Atom token, and some cool new validation software that people are
[01:10.960 --> 01:17.560]  releasing and trying out. I'm also the curator of the cryptocurrency security standard auditor exam,
[01:17.560 --> 01:22.800]  so that's an exam regarding the cryptocurrency security standard that I referenced earlier.
[01:22.800 --> 01:29.340]  I have my own consulting firm where I consult on security and blockchain technology. And then
[01:29.340 --> 01:33.600]  if you guys want to reach me on Twitter, you can hit me up at forwardsecrecy.com.
[01:33.700 --> 01:37.580]  The reason I'm giving this talk to you guys today is because I do have a lot of experience
[01:37.580 --> 01:45.780]  on testnet and mainnet with deploying, configuring, administering, and securing validators
[01:45.780 --> 01:54.170]  and validator infrastructure. I wanted to set up a goal for this talk. So you guys can read it there,
[01:54.170 --> 01:59.810]  I'm not going to go through that. I do want to highlight my redundant joke here. Hopefully you
[01:59.810 --> 02:06.870]  guys like that one. If not, oh well. But my goal for this talk is to have you guys walk away here
[02:06.870 --> 02:12.590]  learning something. There's a lot of smart people here at DEF CON in the blockchain village.
[02:13.130 --> 02:18.030]  But while we may be good at infrastructure, or we may be good at security, or we may be really
[02:18.030 --> 02:23.810]  good at the blockchain aspects of it, nobody's perfect, right? Nobody's an expert in all areas.
[02:23.810 --> 02:28.970]  Hopefully you walk away from this talk taking a tip or some piece of knowledge that you can
[02:29.490 --> 02:32.610]  run with yourself or implement in your own system.
[02:37.100 --> 02:46.580]  So what are validators? Everybody's kind of familiar with the concept of Bitcoin and SHA-256
[02:46.580 --> 02:52.920]  and mining, right? So when you're mining, you're throwing a ton of resources at cryptographic
[02:52.920 --> 02:59.780]  problems to solve them and write data to a blockchain. And mining was awesome. It was very
[02:59.780 --> 03:04.460]  profitable for people for a long time. It helped secure the network. But there's a lot of downside
[03:04.460 --> 03:11.060]  to miners that people have brought up over time, including energy consumption and administration
[03:11.060 --> 03:18.300]  and things like I can't get the newest graphics card to play PUBG or Fortnite because there's
[03:18.300 --> 03:23.840]  run on them, right? There are five, $600 now because everyone's buying them up and mining.
[03:24.480 --> 03:29.460]  So some new coins and chains came out that implemented what was called proof of stake.
[03:29.540 --> 03:34.720]  And unfortunately, it's not the stake that I enjoy every night. I would love a proof of actual
[03:34.720 --> 03:40.420]  like a one type of stake. But this proof of stake is where we're actually staking and bonding coins
[03:40.420 --> 03:47.500]  and doing our network through validators. So those validators actually provide a whole bunch
[03:47.500 --> 03:52.840]  of energy savings, and they can achieve kind of the same end result. They're able to validate
[03:52.840 --> 03:58.960]  transactions, validate blocks, provide signatures, create chain data and do it in a way where it's
[03:58.960 --> 04:04.740]  more beneficial to the environment and it's lower cost and overhead. The way it works, as I said,
[04:04.740 --> 04:11.440]  people end up bonding or locking or staking the coins with the validator. And then for this talk,
[04:11.440 --> 04:15.820]  we're going to be focusing a little bit more on Cosmos. But with things like ETH2,
[04:15.820 --> 04:20.260]  they have some different systems where they kind of bet on blocks and transactions and things for
[04:20.260 --> 04:25.100]  the validators to process. But on the back end, what it's really doing is it's taking cryptographic
[04:25.100 --> 04:36.820]  signatures and signing and broadcasting votes out to the chain. The incentive for the validator,
[04:36.820 --> 04:43.060]  you can see right here is money, basically, digital value. So as a validator, you're validating all
[04:43.060 --> 04:47.360]  these transactions, and you're running your infrastructure and you're putting in all the
[04:47.360 --> 04:52.380]  time and effort. And what's the incentive to go and do this? So there's a few different pieces.
[04:52.380 --> 04:58.000]  The first one's the security aspect. So you're providing back to the security of the chain,
[04:58.000 --> 05:02.720]  the decentralization aspect. So you're participating in a decentralized system.
[05:02.720 --> 05:09.480]  There's a governance aspect where as a validator, if you meet certain conditions based off of the
[05:09.480 --> 05:14.300]  chain and governance that's been set, you can actually participate in governance votes and
[05:14.300 --> 05:20.400]  help to modify and make changes to that chain and its different variables moving forward.
[05:20.760 --> 05:25.320]  But there's also block rewards and transaction fees. And that's kind of where the money comes in
[05:25.320 --> 05:31.620]  at. And this is just a very small example. So I think this is one day of rewards for a validator
[05:31.620 --> 05:36.560]  that I was running on a new chain that was just released. And you can see here, I'm getting some
[05:36.560 --> 05:43.280]  die and I'm getting some tick. And this validator, while it's using tick as its denomination and also
[05:43.280 --> 05:48.800]  paying some commissions in die, this could be whatever coin or whatever chain that you're
[05:48.800 --> 05:54.240]  running your validator for. So in the instance of Cosmos, that would be Atom, right? And today,
[05:54.240 --> 05:59.820]  it's kind of set to a single chain. But the real value for validators is that you can run validators
[05:59.820 --> 06:05.100]  on multiple different chains and tokens and provide the security and the governance and get the
[06:05.100 --> 06:09.400]  kickbacks from that. But in the future, with some of the scaling efforts and new technology that's
[06:09.400 --> 06:13.940]  coming out, those validators are going to be able to validate multiple types of coins.
[06:13.960 --> 06:18.380]  So if I'm a validator that's running a whole bunch of different chains, and I'm validating on that,
[06:18.380 --> 06:25.180]  my rewards for the day could be Bitcoin, Dogecoin, Litecoin, Die, Tick, whatever I choose to validate
[06:25.180 --> 06:30.840]  on or stake my tokens with. And that's something that's going to be important that we're going to
[06:30.840 --> 06:36.040]  talk about a little bit later. But from the validator aspect, like I said, the rewards,
[06:36.040 --> 06:41.060]  transactions, governance, being able to participate in the system. And the other thing that's important
[06:41.060 --> 06:46.020]  is the more stake, the more coins or tokens that have been bonded with that validator,
[06:46.020 --> 06:51.400]  the more power and say that validator has within the system. So it's very important to make sure
[06:51.400 --> 06:56.260]  that you're keeping up on that and that you're bonding those tokens and contributing back to
[06:56.260 --> 07:04.100]  the ecosystem in order to be a good neighbor. The other reason validators run this, so while
[07:04.100 --> 07:08.900]  you're getting block rewards and you're contributing, it's pretty cheap to do this. So this is an
[07:08.900 --> 07:14.120]  unnamed cloud provider. And this is just some simulated costs. So as you can see for the month,
[07:14.120 --> 07:18.640]  it's extremely cheap to run these instances or they can be virtualized instances, they could
[07:18.640 --> 07:25.360]  be bare metal instances. There's a lot of benefits to running up in a cloud provider, which we're
[07:25.360 --> 07:31.260]  going to jump into. But the overall goal of this is it's extremely cheap to do this from an
[07:31.260 --> 07:37.360]  infrastructure side, not as necessarily as cheap from the management side and the personnel, the
[07:37.360 --> 07:42.640]  human cost, but at least getting things spun up, running a validator. You can get this done in an
[07:42.640 --> 07:47.960]  afternoon if you're the type of person that has some familiarity with cryptocurrency, has
[07:47.960 --> 07:53.500]  familiarity with the Linux command line, and has some familiarity with best practices and security.
[07:53.560 --> 07:59.320]  Because you are dealing with things that have a ton of value associated with them.
[08:00.460 --> 08:06.200]  There's a lot of different networks that currently run validators and do these proof of stake type
[08:06.200 --> 08:11.420]  of systems. Cosmos here in the middle is the big one that we're focusing on for this talk.
[08:11.420 --> 08:15.240]  But a lot of the information presented here can be applied out to a lot of these different
[08:15.240 --> 08:21.480]  networks that run validation systems. Microtip down here in the right is a brand new one that
[08:21.480 --> 08:25.540]  just came out within the past couple of weeks, they launched their mainnet. And it's one that
[08:25.540 --> 08:31.540]  I've practiced some validation on. And they're doing some interesting things with price discovery
[08:31.540 --> 08:37.340]  and stuff of assets. And it's really interesting because you're seeing stuff use some of the Cosmos
[08:37.340 --> 08:41.540]  SDK, where a lot of these things that we're talking about are going to be applicable.
[08:41.660 --> 08:45.860]  Other chains are now forking those off, and kind of doing their own things with it,
[08:45.860 --> 08:50.760]  creating great new products and great new services for end users that we can all jump in and play
[08:50.760 --> 08:56.220]  with. So if anybody after this talk does have interest in validating, I would definitely go out
[08:56.220 --> 09:00.720]  to some of these networks, look at their documentation, spin up some virtual instances,
[09:00.720 --> 09:06.420]  or if you have some spare metal spare hardware, try jumping on their test net and seeing
[09:06.420 --> 09:18.820]  how this all works out. With Cosmos specifically, we were talking about value. And we can see here
[09:18.820 --> 09:25.440]  some stats from one of the block explorers for the Cosmos Hub. And right now the atom token,
[09:25.440 --> 09:32.940]  I think, is at 415 at the time of this presentation. It might be 417 now, it might be 413 in a second
[09:32.940 --> 09:38.020]  from now. They've got about 3 million blocks on their current chain. And this is really what's
[09:38.020 --> 09:43.660]  important. So this is showing how many of the atoms are bonded or staked back into the system.
[09:43.660 --> 09:49.760]  Right now it's at 71%. So 71% of all the atoms that are out there are being staked and held
[09:49.760 --> 09:54.720]  securely for these validation services. Now, when we take a look at the current price and
[09:54.720 --> 10:01.640]  we equate that out, that's about $760 million currently that's being bonded behind this network
[10:01.640 --> 10:08.080]  trying to protect it and prop it up. Another important number here is inflation, which you'll
[10:08.080 --> 10:12.000]  see at 7%. Remember that one because we're going to talk about it here in a couple slides.
[10:16.510 --> 10:21.530]  So we now have validators set up in this system that are validating transactions.
[10:21.810 --> 10:28.730]  And these validators could be run by you, they could be run by an entity, or some other third
[10:28.730 --> 10:35.230]  party or trust. Anybody can run a validator. But the other thing you can do as a user is you can
[10:35.230 --> 10:41.410]  stake your own atoms or tokens with a validator. And the reason you would want to do that is to
[10:41.410 --> 10:46.630]  additionally profit. So this is kind of similar to the whole staking pool concept with mining,
[10:46.630 --> 10:52.790]  where I don't want to run all the hardware and I don't want to take on the risk. But I do want to
[10:52.790 --> 10:57.790]  contribute and benefit in some way. And this is a great way that anybody can go and participate
[10:57.790 --> 11:02.810]  in these networks and contribute to these validators, and also potentially profit and
[11:02.810 --> 11:09.330]  reward from it at the same time for contributing to that network. There is some things with
[11:09.330 --> 11:14.810]  validators where they're going to get block rewards. And the reason you would want to risk
[11:14.810 --> 11:20.190]  your money with a validator is that block reward, there's an inflation component here. And that was
[11:20.190 --> 11:26.670]  the 7% that we were talking about. So there's an incentive for validators and holders to stake
[11:26.670 --> 11:33.390]  their tokens over time. You can see here the average yield in this example is 9.46%. So if
[11:33.390 --> 11:41.550]  you compare that to a 7% inflation rate, you're up to 2.4% at that point. So it still makes sense
[11:41.550 --> 11:45.730]  to bond tokens into this. And as long as that average stays above the inflation rate on the
[11:45.730 --> 11:52.510]  network, it's profitable for you to be bonding your tokens over time and getting those interest
[11:53.050 --> 11:57.670]  tokens back, essentially. So you would think of this as kind of like a savings account or a high
[11:58.150 --> 12:03.810]  account where you have to manually go back in and re-delegate your tokens to a validator,
[12:03.810 --> 12:08.390]  pull out your rewards and re-delegate them. And then over time, your reward amount is going to
[12:08.390 --> 12:14.750]  grow. And just some statistics here. So for 686 atoms monthly, you would be getting a return of
[12:14.750 --> 12:23.290]  5.41. So at $4, you're looking at $20, $25 for basically just doing passive income at that point.
[12:28.020 --> 12:34.220]  Now, when it comes to validators, another thing that incents them outside of the inflation
[12:34.220 --> 12:38.480]  component is slashing. And when we're talking about this, we're not talking about the guy from
[12:38.480 --> 12:45.620]  Guns N' Roses. We're actually talking about penalizing participants in the network for
[12:45.620 --> 12:51.320]  violating certain conditions. And this can be in the network, this could be in the consensus layer.
[12:51.320 --> 12:56.160]  But it's a huge financial incentive for validators to play nice because
[12:56.160 --> 13:00.760]  anytime you introduce these types of systems, you're going to have hackers, you're going to
[13:00.760 --> 13:05.000]  have bad actors, you're going to have people trying to gamify it and say, you know, how can
[13:05.000 --> 13:10.280]  I increase my rewards? How can I knock you offline? How can I trick the system or the state of the
[13:10.280 --> 13:17.480]  system to go in there and profit for me? What's the incentivization to not do that? And slashing
[13:17.480 --> 13:23.160]  is a great component for that. And there's two ways that you can really get slashed when you're
[13:23.160 --> 13:31.020]  talking about Cosmos and proof of stake coins currently. They are downtime and double sign.
[13:31.020 --> 13:36.920]  Both start with D, pretty easy to remember. Downtime has some conditions where out of the
[13:36.920 --> 13:43.880]  last 10,000 blocks, if you're down for more than 95% of that, you're risking a downtime slashing
[13:43.880 --> 13:49.460]  event. You can see here in this block explorer, these blocks were being sent and everything was
[13:49.460 --> 13:55.560]  great. And then at some point, something went offline and now we're missing them. So if that
[13:55.560 --> 14:02.820]  continued for the next 9,500 blocks or whatever, and we are not able to get our validator troubleshot
[14:02.820 --> 14:08.500]  or figure out what's going on or move to another system, we risk incurring a downtime event and
[14:09.360 --> 14:14.640]  having a slashing event occur. The other big one is a double sign.
[14:14.780 --> 14:19.600]  And this is an example of a double sign that happened on the Cosmos mainnet where what ended
[14:19.600 --> 14:27.160]  up happening was the private key for the validator was used to sign two transactions,
[14:27.160 --> 14:32.760]  sign the votes for them for the same block. And that was pushed out, that was detected.
[14:32.760 --> 14:38.440]  And as a result, that validator was jailed because you can't double sign on your vote.
[14:39.260 --> 14:44.820]  So once you're jailed for a double sign, unfortunately, you are in jail. There is
[14:44.820 --> 14:48.940]  no getting out. Your validator will not be able to rejoin the system.
[14:49.660 --> 14:52.460]  You've screwed up and you've screwed up in a major way.
[14:55.770 --> 15:02.270]  When it comes to downtime, the amount of stake that can be slashed is very minimal. It's not
[15:02.410 --> 15:07.770]  a huge amount, but depending on how many atoms are staked in your validator by yourself as the
[15:07.770 --> 15:14.570]  validator, what you're doing is called self-staking. So a validator may have a million
[15:14.570 --> 15:19.130]  atoms and stake their million atoms to say, I believe in my system, I believe in myself,
[15:19.130 --> 15:22.750]  I'm going to stake all my atoms with it. But as I said, users can also stake.
[15:22.930 --> 15:26.930]  So when these things, when these slashing events happen, not only is the self-stake affected,
[15:26.930 --> 15:31.850]  but the user stake can be affected too. So as a validator, if you do have a slashing event and
[15:31.850 --> 15:36.830]  you lose stake, you're losing money for your end users, tokens, and digital value for your
[15:36.830 --> 15:42.410]  end users. They're not going to be very happy with you or want to contribute to you in the future.
[15:43.630 --> 15:48.330]  The penalty for that, you miss your rewards while you're in a... and that goes for a 10 minute
[15:48.330 --> 15:55.390]  period. And with these types of events, usually you only get hit for the first one. It's not
[15:56.250 --> 16:02.010]  reoccurring. They put, I think it's called a tombstone. I know there's a tombstone here for
[16:02.010 --> 16:06.510]  double signing, but I forget what it's called for downtime. I apologize. But it's basically kind of
[16:06.510 --> 16:12.630]  like a buffer to say that we hit you for downtime. We're not going to hit you again if you're in a
[16:12.630 --> 16:18.210]  certain period. So if you are being slashed for downtime because your server's down or you're
[16:18.210 --> 16:24.470]  having some sort of redundancy issue, it's still within your best interest to get that up as soon
[16:24.470 --> 16:29.850]  as possible because these things are going to start to get costly over time. But there is some
[16:29.850 --> 16:33.770]  protection there for end users so that it's not just going to consistently stack over time unless
[16:33.770 --> 16:40.310]  certain conditions are met. With double signing, it's huge. It's 5%. So if you do have a million
[16:40.310 --> 16:46.270]  dollars in Atom on your validator and taking a 5% slash stake, you guys can figure that out and see
[16:46.270 --> 16:51.410]  how much money that is, right? You also hit an automatic unbonding penalty. If you double sign,
[16:51.410 --> 16:57.110]  and it also incurs a three-week unbonding period where those Atoms cannot be touched while they're
[16:57.110 --> 17:03.450]  being unbonded from the validator. I put the little tombstone here for the RIP because if
[17:03.450 --> 17:07.770]  you double sign, you're done. You're not going to be able to move forward with that validator.
[17:07.770 --> 17:14.130]  You're either going to need to re-delegate all of your stake to another validator or allow your
[17:14.130 --> 17:21.390]  users to unbond all their stake and move forward with somebody else at that point. You're out.
[17:25.660 --> 17:30.060]  Because of all this, you want to make sure you're doing your due diligence when you're either
[17:30.980 --> 17:35.900]  staking with a validator or thinking about running one yourself. These are all points
[17:35.900 --> 17:41.040]  you should be thinking about. So it's recommended from my experience that you want to diversify
[17:41.040 --> 17:46.740]  at minimum probably three is a good bet. I don't want to say best practice. I usually like to stick
[17:46.740 --> 17:51.960]  the three to five different validators for different tokens. The reason being is that
[17:51.960 --> 17:56.740]  we're in a decentralized system and we should be decentralizing as much as possible. I'm also
[17:56.740 --> 18:01.820]  mitigating my risk. So if one of those three or one of those five validators do incur a slashing
[18:01.820 --> 18:06.660]  event or a downtime event for whatever reason, I've mitigated my risks where I'm not losing all
[18:06.660 --> 18:13.520]  of my funds at the same time. You want to do some open source research, some open source intelligence
[18:13.520 --> 18:19.420]  and look up the teams behind them. Look up the history of the validator. So what does their
[18:19.420 --> 18:25.260]  history look like? Is it clean? Do they have a lot of downtime or maybe a misconfiguration where every
[18:25.260 --> 18:31.920]  99 out of 100 blocks they're not signing for some reason? Well, why is nobody noticing that
[18:31.920 --> 18:37.460]  or fixing that? Are they really paying attention to their system? Have they been jailed before? So
[18:37.460 --> 18:43.960]  it is possible to go to jail and get out of jail. As we said, if you double sign, you're done.
[18:44.120 --> 18:50.540]  But for minor events, you can get out of jail if you accept, if you levy the penalty at that point
[18:50.540 --> 18:56.820]  and you wait for the time that you're jailed. So if they have a history of jail occurrences that
[18:56.820 --> 19:00.800]  you're not comfortable with, you shouldn't be giving them your money because this is essentially
[19:00.800 --> 19:05.720]  what you're doing is you're letting them hold digital value on your behalf, essentially.
[19:06.620 --> 19:11.520]  And then the other thing is a lot of teams have gotten really good in the past year with
[19:12.000 --> 19:16.760]  putting a presence out there. So there's people and there's corporations that are doing this as
[19:16.880 --> 19:22.200]  a business, right, where they validate on all the chains I listed earlier and more,
[19:22.200 --> 19:26.520]  because there's a ton of them out there doing their own little niche with different things
[19:26.520 --> 19:31.440]  with proof of stake and their chains and their use case. But some of those teams are getting
[19:31.440 --> 19:36.340]  really good with publishing a social presence. So they're listing their teams on web pages,
[19:36.340 --> 19:39.700]  or they're showing you with chains, or they're actually showing you pictures of
[19:39.700 --> 19:45.440]  here's what our setup looks like, our validator setup looks like. Be careful when you do that,
[19:45.440 --> 19:50.640]  because it's very similar to the ICO craze of 2017, where some of the team pictures,
[19:50.640 --> 19:56.660]  if you go and reverse Google search or Tanai reverse search those, you're going to find that
[19:56.660 --> 20:01.920]  those are some LinkedIn picture for somebody that works in like, you know, aviation or something
[20:01.920 --> 20:05.960]  like that, that has nothing to do, and they're just stealing your information to try to fleece
[20:05.960 --> 20:12.380]  you. So do your due diligence, don't get scammed. The reason you want to do that.
[20:12.380 --> 20:21.560]  So this is a personal example. I was validating on a testnet for a coin that was not released yet.
[20:21.920 --> 20:28.360]  And because it was a testnet, and because I wasn't taking it as seriously as I should have,
[20:28.360 --> 20:37.800]  I ended up using some spare hardware, and ended up experiencing like a 36 hour power outage.
[20:37.800 --> 20:44.480]  This spare hardware was hooked up to a Comcast cable modem, no redundancy across the board.
[20:45.040 --> 20:50.320]  And during the power outage, because I'm focusing on saving my food and a toddler,
[20:50.320 --> 20:55.160]  and what are we going to do, supposed to be 95 degrees tomorrow, and all of those things that
[20:55.160 --> 21:02.500]  you have to deal with when life hits you and kind of shit hits the fan. Those are the things that
[21:02.500 --> 21:05.680]  you're not going to be thinking about your validator, right? You're not going to be thinking
[21:05.680 --> 21:09.840]  about the thing that you've set up in the dressing room that just runs off the modem that
[21:09.840 --> 21:18.080]  your wife doesn't like, or your kid resets all the time. So unfortunately, I'm a victim of this.
[21:18.080 --> 21:23.000]  And thankfully, it was only on testnet, but I ended up losing 10,000 coins because I did not
[21:23.000 --> 21:27.820]  have a redundant infrastructure set up. Now, if that was to occur on mainnet, and I would
[21:27.820 --> 21:32.620]  say a validator, I would have just taken a big PR hit and lost my users funds.
[21:32.620 --> 21:37.120]  In my instance, thankfully, it was only testnet, and it was a great lesson learned.
[21:38.500 --> 21:44.420]  And this is actually the picture of my validator. So this was an Intel NUC that I had a spare
[21:44.420 --> 21:49.420]  machine that I set up, and I said hooked it to a cable modem with a single Ethernet line,
[21:49.420 --> 21:56.720]  single power line, right, single SSD in here somewhere. So no redundancy, no infrastructure,
[21:56.720 --> 22:01.200]  not even really any status lights to tell me what's going on. So if you are thinking about
[22:01.200 --> 22:07.460]  running a validator and you find that the people you're thinking about bonding your tokens with
[22:07.460 --> 22:13.280]  are doing this, you should run away as fast as possible. Great for development and great
[22:13.280 --> 22:18.400]  for testing, should never be done in production. That's what you want to do in production.
[22:18.400 --> 22:23.040]  So we want to see the data center, we want to see the cool blinky lights. The reason you want
[22:23.040 --> 22:30.220]  to do this is you're getting cooling here, you're getting heat and power and network and all the
[22:30.220 --> 22:33.920]  good warm fuzzy feelings that go into your hosting bill when you have to pay that at the end of the
[22:33.920 --> 22:40.440]  month to justify all the servers and things that are being filled in these racks. So you know,
[22:40.440 --> 22:44.840]  stand on the shoulder of giants, use the experts, use the redundant infrastructures that have
[22:44.840 --> 22:49.520]  already been built and hopefully been tested correctly to host these types of things and
[22:49.520 --> 22:58.810]  do validation. We want to implement all like the standard best practices.
[22:59.830 --> 23:03.330]  When it comes to redundancy, you want to hit all the...
[23:05.290 --> 23:09.290]  Monitoring, we're going to jump into with a couple different solutions that can actually
[23:09.290 --> 23:15.890]  alert you when you do have the 36-hour power outage. There's a lot of administration concerns
[23:15.890 --> 23:22.930]  too with different configurations and things like Cosmos has published a security update for
[23:22.930 --> 23:29.510]  its SDK and it's an emergency upgrade. How are we going to do that, right? Do you test the data
[23:29.510 --> 23:34.890]  center tech with going in and upgrading that binary? Or do you have a subject matter expert
[23:34.890 --> 23:40.070]  on staff that can handle that? Do they know the difference between go path and your environment
[23:40.070 --> 23:44.810]  variable path and all of those types of things? Because all of those are considerations that
[23:44.810 --> 23:49.330]  you're going to hit and speed bumps that are going to occur while you're working on these
[23:49.330 --> 23:55.110]  types of systems. You need something to do some sort of repo monitoring, be able to check when
[23:55.110 --> 23:59.970]  upgrades and new checkpoints and releases are released. And then deployment is another big
[23:59.970 --> 24:09.230]  consideration for administration. There's a lot of different schools of thought on deployment.
[24:09.450 --> 24:16.050]  And we'll jump into some of the use cases as to why some are better than the others.
[24:16.050 --> 24:21.450]  But it's definitely a concern when you're architecting and administering these types of systems.
[24:21.550 --> 24:24.750]  And then we want to jump into a lot of the security components with
[24:24.750 --> 24:28.890]  controls and the different aspects on how you're mitigating your risk and how you're
[24:28.890 --> 24:33.330]  delegating access and who knows what and why do they need to know it.
[24:37.510 --> 24:42.850]  The keys that actually work on a validator when we're working on security, there's two keys specifically for
[24:42.850 --> 24:47.070]  Cosmos that need to be protected and thought about with all those prior best practices
[24:47.070 --> 24:52.970]  for redundancy and backups and things of that nature. And that's your tenderment key, which is
[24:52.970 --> 24:58.570]  actually doing your consensus voting. And then the application key, which derives your validator
[24:58.570 --> 25:03.770]  addresses and things. That signs your actual transactions. But both of those need to be
[25:03.770 --> 25:09.810]  considered when you're building these systems on, one, where does it live? Two, how did we generate
[25:09.810 --> 25:15.810]  those? Did we do it in a secure way? And I'll show you some ways that you guys can do that.
[25:15.810 --> 25:23.970]  Three, where are we backing it up? So does it just exist on some EC2 instance somewhere? Or do I have
[25:23.970 --> 25:31.910]  key material written down in a waterproof, fireproof safe that's access control? A lot of
[25:31.910 --> 25:36.030]  that stuff needs to go into the thinking and mindset and paradigm of how you're protecting
[25:36.030 --> 25:40.550]  the system and how you're doing it. And you really want to do it right kind of from the beginning,
[25:40.550 --> 25:47.090]  because if you deploy a validator to determine that I didn't generate my key material securely,
[25:47.090 --> 25:52.210]  you don't want to have to go back and redo all that stuff and re-delegate and try to make sure
[25:52.210 --> 25:56.110]  that people are aware that, you know, we're validator two instead of validator one now,
[25:56.110 --> 26:01.170]  those types of things. So when you're building this stuff, kind of encompass all this in your
[26:01.170 --> 26:05.770]  thinking and do it right from the beginning. Because really what these keys are protecting,
[26:05.770 --> 26:11.290]  I checked CoinCap IO right before this presentation, 1.3 billion market cap for
[26:11.290 --> 26:18.090]  Cosmos Atoms right now. So for a coin that's still relatively brand new in the scheme of things,
[26:19.090 --> 26:24.170]  we're already at a billion dollar market cap at current day prices. So if the bull market really,
[26:24.170 --> 26:30.410]  really starts taking off, at that point, these keys are protecting crazy amounts of value. Are
[26:30.410 --> 26:36.570]  you comfortable leaving that type of information sitting on a server that, you know, the data
[26:36.570 --> 26:41.510]  center tech down the street that happened to work on the day that you called in and said,
[26:41.510 --> 26:45.990]  my server's down, can you check the lights on my Dell system? You know, he's looking at console
[26:45.990 --> 26:52.550]  and he's single usering and things. While your key material may be protected by local key stores
[26:52.550 --> 26:57.390]  and things like that, what stops him from sniffing memory? What stops him from imaging the drive or
[26:57.390 --> 27:02.310]  breaking RAID or things like that and then clearing logs? So it's considerations that you
[27:02.310 --> 27:08.450]  want to think about when you're building those systems. For redundancy, it's all the great stuff
[27:08.450 --> 27:15.590]  with power supplies and phasing and networking. And I've seen some validators where they run on
[27:15.710 --> 27:21.390]  a single networking. So what happens if that network card fails or the cable goes bad,
[27:21.710 --> 27:26.670]  a pair in that cable goes bad? You're going to be at risk of a potential downtime event.
[27:26.670 --> 27:32.990]  When it comes to storage, there is a lot of data that goes into running a validator. A validator
[27:32.990 --> 27:39.870]  essentially needs on itself a copy of the full node, the chain data, or it needs to link to
[27:39.870 --> 27:46.930]  another server a century that has a copy of that full node chain data. So if you're building in
[27:46.930 --> 27:52.670]  redundancy to say if this instance goes down, once you rebuild your new instance with your new
[27:52.670 --> 27:58.290]  validator binaries, you need to sync all the chain data again. So if you're in that window,
[27:58.290 --> 28:03.430]  that downtime window, that 95% of 10,000 blocks, which I think is like half a day,
[28:03.430 --> 28:08.170]  and it's starting to get down to the wire, I've seen where we've started binaries and things like
[28:08.170 --> 28:13.150]  that and my validators, you know, they still need two hours to catch up to sync chain data
[28:13.150 --> 28:17.610]  because I didn't have a copy of it anyway. So I would definitely recommend looking at things like
[28:17.610 --> 28:24.650]  RAID if you're using SANs, doing snapshots and things of that nature. The Cosmos team has
[28:24.650 --> 28:30.350]  published snapshots over time of chain data where you can go and download those types of things to
[28:30.350 --> 28:35.610]  sync it, but I'd be careful with what sources you're pulling that from because if a bad actor
[28:35.610 --> 28:40.250]  wants to cede bad data to you, you could potentially risk another issue with your validator at that
[28:40.250 --> 28:44.810]  point. And then there's other some some other gotchas that people don't necessarily think about
[28:44.810 --> 28:49.290]  and like, there's some funny stories and some instances, you're standing in front of the
[28:49.290 --> 28:56.630]  server in the data center, and you're going, I'm here. I have no way to log in, there's no keyboard,
[28:56.630 --> 29:01.190]  right? There's no crash cart available in the data center. So run through some of your BCDR
[29:01.190 --> 29:06.650]  type of policies with like, what do we need in the cage, right? Do we need spare hard drives? Do we
[29:06.650 --> 29:12.590]  need installation software for OSs if we have to spin it up? Do we need config variables to point
[29:12.590 --> 29:17.450]  to a PXE server or to point to a Kubernetes cluster? However you decide to architect your
[29:17.450 --> 29:23.650]  solution that best works for your information system. And I note down here, any time that you're
[29:23.650 --> 29:28.330]  running through this type of stuff and architecting it and running through these exercises, you always
[29:28.330 --> 29:33.430]  want to think about the worst case scenario, which is double signing. And in order to achieve that
[29:33.430 --> 29:39.230]  double sign, it means that those Cosmos keys need to be either duplicated somewhere or pulled into
[29:39.370 --> 29:44.610]  a configuration or have two instances that are running that same material. So always think about
[29:44.610 --> 29:49.710]  that. That's kind of the grenade in the mix that needs to always move forward and can never be
[29:49.710 --> 29:59.770]  copied because if it is and it goes live, you're risking that 5%. I put this slide in here just
[30:00.270 --> 30:07.170]  because I hate DNS. DNS is a big one. And you want to be careful with this because when you
[30:07.170 --> 30:11.970]  start to build validator infrastructure and you start to do additional security controls and
[30:11.970 --> 30:18.550]  different servers with copies of chain data, DNS is one that may get you that you don't even realize
[30:18.550 --> 30:23.790]  because nobody ever thinks it's DNS, but it always is. So make sure your host names are correct.
[30:23.790 --> 30:30.050]  Make sure you have redundancy upstream in your network stack, not just the cables, even the
[30:30.050 --> 30:36.870]  routers in the rack. If you're using things like Netgear or Cisco equipment, do you have that in
[30:36.990 --> 30:43.070]  a stack configuration? Do you have an AB failover? If your switch or your Cisco ASA license expires
[30:43.070 --> 30:48.510]  and you start getting throttling or your feature stops off, what happens into your system? So those
[30:48.510 --> 30:52.750]  are all things to simulate and think about when you're building that. And DNS is one of the ones
[30:52.750 --> 31:02.120]  that I don't want people to forget about. So now we need to monitor all this stuff with our
[31:02.660 --> 31:08.600]  validators. We've figured out how it works. We've got some coins. We've got a system set up.
[31:08.600 --> 31:15.640]  Now, how do we avoid that 36-hour outage? So SimpliVC is a company that's out there that
[31:15.640 --> 31:20.400]  does a ton of validation on chains and things like that. And they release this open source software
[31:20.400 --> 31:25.780]  called Panic. And I'm a big, big fan of open source software. But what this will do is this
[31:25.780 --> 31:31.900]  will actually monitor things from the tendermint layer and from the local server layer. And it'll
[31:31.900 --> 31:39.440]  do email, telegram, through Twilio integrations, it's not responding. And it'll also do the repo
[31:39.440 --> 31:44.940]  monitoring we talked about earlier. You know, a new version of Cosmos, okay, or Gaia-D or whatever
[31:44.940 --> 31:49.560]  binary has been released, you need to be aware of that so that you can go check the release notes
[31:49.560 --> 31:56.040]  and see if this is something I need to apply now or can I defer that to later? I love it. I use it
[31:56.040 --> 32:01.840]  for all my validation stuff. This is my telegram integration. And you can see it doesn't just
[32:01.900 --> 32:08.700]  do things like the services down. It actually tells things like my peers have increased or the
[32:08.700 --> 32:14.400]  node was inaccessible. If people delegate to my validators, it gets updates on my voting power.
[32:14.400 --> 32:18.420]  So I know that I've got more power in the system for consensus votes or that there's something
[32:18.420 --> 32:22.840]  happening. And I need to be looking at my validator because people are delegating to me.
[32:22.840 --> 32:27.160]  I want to know why. And then as I said, it gives you peering changes. So if other people are doing
[32:27.900 --> 32:32.560]  peers dropping off the network, that may be indicative of a network event or a binary upgrade
[32:32.560 --> 32:36.340]  or something else is going on with those other servers. They're doing some work to that stuff.
[32:36.340 --> 32:40.720]  And it's good to be aware because we may need to take mitigation efforts.
[32:43.450 --> 32:48.170]  Prometheus is another great one. This is all packaged with the current Gaia installation
[32:48.170 --> 32:54.490]  with Cosmos. It's a very simple config change. And this is basically kind of like a logging port
[32:54.490 --> 33:00.630]  that opens up a whole bunch of data that's going on in your validator that you can feed into Elk
[33:00.630 --> 33:05.910]  stack or different dashboards or whatever you choose. But it has really good support for Grafana,
[33:05.910 --> 33:12.330]  for example. And this brings it into the kind of DevOps infrastructure world of,
[33:12.330 --> 33:16.630]  I want to build dashboards and I want cool graphs and stats and stuff like that.
[33:16.850 --> 33:21.710]  So if you don't want to do that through necessarily a block explorer, if you want to run something
[33:21.710 --> 33:26.550]  locally, these things are, it's great to turn Prometheus on. It's great to throw Grafana or
[33:26.550 --> 33:35.020]  some sort of Elk stack monitoring on top. So we've got our monitoring taken care of and now we're...
[33:37.180 --> 33:44.680]  So we talked about the data center tech at, let's use AWS, for example. So you're running
[33:44.820 --> 33:49.980]  a validator in AWS and you've got this private key material on there that's validating transactions
[33:49.980 --> 33:55.480]  and participating. What stops the data center tech from logging into your system and grabbing
[33:55.480 --> 33:59.540]  your key material? What stops the hacker from getting on your server and grabbing that stuff
[33:59.540 --> 34:04.900]  remotely if you haven't configured your security groups and your firewalls correctly? So that's
[34:04.900 --> 34:10.340]  the problem and the solution is HSMs. And HSMs are great if you guys have never used them. They're a
[34:10.340 --> 34:15.080]  little bit expensive, but it's very similar to the way that, you know, YubiKeys and things like that
[34:15.080 --> 34:20.940]  where it stores the private key material actually on a hardware device and then signs messages
[34:20.940 --> 34:26.540]  without actually having to expose or reveal that material. In theory, it's not supposed to allow
[34:26.540 --> 34:31.560]  any private key extraction. Now there have been attacks and that's not to say that things can't
[34:31.560 --> 34:37.180]  fail in the future, but currently it's not supposed to allow any private key extraction.
[34:37.420 --> 34:42.840]  So what you would do is when you were generating key material to either set up your validator or
[34:42.840 --> 34:48.320]  your wallet or functioning, you want to generate those keys in a secure way and then store that
[34:48.320 --> 34:54.140]  private key material on a hardware device. The two I've used is the YubiHSM2 and the Ledger Nano S,
[34:54.140 --> 34:59.980]  which both work very well with Cosmos. I know they work with MicroTik and they work with a few other
[34:59.980 --> 35:05.460]  other chains that are out there. But this would actually physically sit into your bare metal
[35:05.460 --> 35:12.100]  server in the data center, in my case, in the dressing room and provide the private key material
[35:12.100 --> 35:17.100]  and signing functions without revealing it. So if somebody was to get into your server remotely,
[35:17.100 --> 35:21.500]  or if the data center texting their single user in, they're not going to be able to extract this
[35:21.500 --> 35:26.940]  without a whole bunch of different authentication information and, you know, secrets and things that
[35:26.940 --> 35:32.320]  they don't have. And then the actual physical hardware security capability of the device too.
[35:33.180 --> 35:40.120]  One thing I will note to be very careful with this stuff is that the YubiHSM in particular
[35:40.120 --> 35:44.700]  has a very small profile and in order to firmware reset this device, I think it's like hold this
[35:44.700 --> 35:49.600]  front side for 10 seconds or something. So if you're in a data center working and you're
[35:50.560 --> 35:56.300]  failing over from server A to server B because you're doing maintenance and you pop your HSM out,
[35:56.300 --> 35:59.940]  be careful how long you're holding it and where you're touching it on. Because if you wipe that
[35:59.940 --> 36:04.680]  private key material, that's not going to be recoverable and you better hope to hell you have
[36:04.780 --> 36:09.580]  a backup. The Ledger works really great with it too and it's got the little graphical display and
[36:10.120 --> 36:14.400]  same thing, you know, there's always security vulnerabilities out for hardware wallets but
[36:14.400 --> 36:19.020]  people are keeping up on those and they're auditing them and it's still a better layer
[36:19.020 --> 36:30.020]  of defense than just leaving the key material on the server itself. Another thing you want to do
[36:30.020 --> 36:36.740]  from a security aspect is you want to set up Sentry architecture. And a Sentry is basically a
[36:36.740 --> 36:43.260]  copy, a full node of the chain that you're validating on. So you could run your validator
[36:43.260 --> 36:48.520]  straight out to the internet and start broadcasting these things. But it's better
[36:48.520 --> 36:55.160]  to put it behind another layer of obfuscation and network connectivity here. And what we're doing
[36:55.160 --> 37:02.760]  is we've connected public internet that was gossiping a lot of information about our transaction
[37:02.760 --> 37:08.600]  validator itself and we've moved it up a layer behind a Sentry layer. And when we do that,
[37:08.600 --> 37:13.380]  this actually enables us to now load balance and have some redundancy at the Sentry layer
[37:13.380 --> 37:17.860]  where we can deploy multiple Sentries here. So I've used two in this example, but it can really
[37:17.860 --> 37:23.480]  scale. I don't know if there's any technical limits on that. That would be a great follow-up.
[37:23.480 --> 37:27.640]  But as far as I'm aware, I haven't seen a technical limit on how many Sentries you can run.
[37:28.140 --> 37:33.020]  But the whole point of this is that nobody's going to be able to get to your validator directly to
[37:33.020 --> 37:37.560]  any public IPs or anything like that. You're going to have private communication between your
[37:37.560 --> 37:44.380]  validator and your Sentry itself. And I've seen this done many different ways where this validator
[37:44.380 --> 37:49.740]  could be inside its own VPC and you have IPsec tunnels talking out to your Sentries, which are
[37:49.740 --> 37:55.860]  in its own VPC that consists of two or three instances. This could just be two or three EC2
[37:55.860 --> 38:02.860]  instances that are talking to each other through private IP space. This validator may not even be
[38:02.860 --> 38:07.780]  the same location or site. It could be bare metal talking up to cloud Sentries somewhere,
[38:07.780 --> 38:13.160]  bare metal Sentries in another data center across the country. I wouldn't recommend that for time
[38:13.160 --> 38:19.800]  to live and packet communication, but it is technically possible. This stops people from
[38:19.800 --> 38:24.860]  denial of servicing your validator. It also gives you some redundancy here for failures with your
[38:24.860 --> 38:29.080]  different Sentries and things like that. And as I said, it protects your communications between
[38:29.080 --> 38:34.260]  your validator and your Sentry. So the validator would get the information from Sentry, sign
[38:34.260 --> 38:40.480]  transactions, broadcast it out through the Sentry out to the public internet. So from an attacker
[38:40.480 --> 38:44.860]  standpoint, they would only get as far as this layer. They wouldn't get to the stuff that actually
[38:44.860 --> 38:56.280]  matters. Other considerations with your security architecture. We never want to double sign.
[38:57.420 --> 39:03.060]  A lot of cloud providers offer auto scaling, and I think that's great for Sentries, but I would
[39:03.060 --> 39:09.560]  avoid it for my validator infrastructure, because you start to get into instances of where does my
[39:09.560 --> 39:15.460]  key material live and how do I migrate? And if server A still has it active and server B spins
[39:15.460 --> 39:20.360]  up and I've copied my key material and that service starts, I've now double signed, right? And I'm in
[39:20.360 --> 39:27.460]  it's game over. Any automation and orchestration pieces with the Sentry, you want to think about
[39:27.460 --> 39:32.720]  the blockchain data and the state data. How are you moving that? How are you verifying that?
[39:32.720 --> 39:36.640]  How are you loading that into new instances if you're doing auto scaling or automation?
[39:36.760 --> 39:41.520]  Or do your new Sentries have to come up and sync the entire chain before they can be active?
[39:42.180 --> 39:46.560]  You also need to worry about doing your config updates. So if I had to have a validator
[39:46.560 --> 39:51.300]  infrastructure and I have two Sentries and add a third, well now I need to go back to all of my
[39:51.300 --> 39:57.280]  existing systems and do some config updates so that I'm now listing that here in my config to say
[39:57.280 --> 40:02.500]  talk to Sentry 3. Don't just talk to Sentry 1 and 2. So when you're building the automation,
[40:02.500 --> 40:06.440]  if you have Sentries and things in that infrastructure, you need to kind of backfill
[40:06.440 --> 40:11.760]  all of those changes to stuff that you've already implemented. And then Sentries are just like
[40:11.760 --> 40:16.260]  validators because it's a full node. If there's security updates for the binary,
[40:16.260 --> 40:20.280]  if there's issues with it where it goes down, you need to be doing all the same
[40:20.280 --> 40:26.060]  all the same repo stuff, all the same binary upgrades when it comes to those Sentries. So
[40:26.060 --> 40:31.540]  if you're scaling your Sentry architecture out horizontally to 40 Sentries, just realize you
[40:31.540 --> 40:39.010]  need to do updates to all of those Sentries when things happen. Then there's some final
[40:39.010 --> 40:46.150]  other considerations here. As I said earlier, there's a lot of different mindsets on should
[40:46.150 --> 40:55.410]  you virtualize validators? And I've gone back and forth on this and I feel like validators should
[40:55.410 --> 41:00.670]  be bare metal. And while it is technically possible to deploy like an EC2 instance or
[41:00.670 --> 41:06.690]  Google's GCP instance and have it validate with key material on that validator, that's not going
[41:06.690 --> 41:11.670]  to be the most secure, optimized, redundant solution that's out there. And if I was considering
[41:11.670 --> 41:17.170]  and looking at two validators on where I wanted the funds, I would not pick that one. I would
[41:17.170 --> 41:20.670]  go with the person that considered some of this stuff. And even if they were using one or two
[41:20.670 --> 41:26.210]  bare metal servers in like a hot or cold fail, to me, I'm a little bit more comfortable with that
[41:26.210 --> 41:31.250]  than somebody that's doing pods of validators that's moving key material back and forth,
[41:31.250 --> 41:35.550]  because all that needs to happen is a weird edge case or anomaly where you're going to hit some
[41:35.550 --> 41:41.450]  sort of double signing scenario, right? And unless you have your deploy scripts tweaked so good with
[41:41.450 --> 41:47.170]  so many tests and error catching and then, yeah, it's scary when you look at the amount of money
[41:47.170 --> 41:53.810]  with that 5% double slash. There's other things you want to consider with minimum config values.
[41:53.810 --> 41:59.590]  So I've done some attacks and I've seen some defense on sentries and things with spam dust
[41:59.590 --> 42:04.870]  transactions where people can try to tax your validator, your sentry resources. And the reason
[42:04.870 --> 42:10.150]  they would do that is to try to edge because maybe they run their own validator or you were
[42:10.150 --> 42:15.930]  jailed and they had money with you or tokens with you, some sort of value. So there's some minimum
[42:15.930 --> 42:20.990]  config values that validators can tweak and sentries can tweak to say, ignore more of the spam
[42:20.990 --> 42:26.150]  and only give me the good stuff. There's an art to it and you got to be careful
[42:26.150 --> 42:30.510]  because you want to be a good system. But just know that you can get pretty granular in your
[42:30.510 --> 42:34.950]  validator config with all the different values that are participating at the tendermint layer
[42:34.950 --> 42:40.610]  and the consensus layer. And that goes for denial service too. And then this last one here that I
[42:40.610 --> 42:45.870]  don't think a lot of people... and I've actually seen a lot of wallet software and validator
[42:45.870 --> 42:52.970]  software getting better with this. People get in the habit of validating and starting to get rewards
[42:52.970 --> 42:59.510]  and they'll pull out all of their rewards and say, re-delegate every single token I have,
[42:59.510 --> 43:04.510]  all my rewards in my wallet, send it back to the validator because I want that compounding interest.
[43:05.310 --> 43:09.050]  And what'll end up happening is it'll work. They'll still sign the transaction and send it
[43:09.050 --> 43:13.910]  out to the validator. And then they literally have zero in their wallet. So you guys, you know,
[43:13.910 --> 43:18.730]  you're familiar with ETH and other systems. If you send all your ETH out and you try to do an ERC20
[43:18.730 --> 43:22.290]  transfer, what's it going to do? It's going to complain and say, you don't have enough ETH for
[43:22.290 --> 43:26.730]  the gas for the transaction. So be careful with that because if you re-delegate all your rewards
[43:27.350 --> 43:31.690]  thing that's in that wallet, you're not going to have enough gas to do any more transactions at
[43:31.690 --> 43:35.350]  that point in the future. So you're going to have to go procure atoms from somebody else or wait
[43:35.350 --> 43:39.530]  until you have enough rewards to get enough to be able to start to continue to do transactions.
[43:39.530 --> 43:45.110]  So I've gotten in the habit of like leaving half an atom or an atom or whatever denomination or
[43:45.110 --> 43:48.870]  whatever token I'm using, leaving one or two of those in the wallet, just so I know that I'm
[43:48.870 --> 43:52.790]  always going to be covered for transactions in the future. And that stuff's getting way better
[43:52.790 --> 43:58.410]  with doing calculations and fees and even notifying users. Cosmos station is a mobile
[43:58.410 --> 44:02.290]  wallet that I love. And it'll come up and actually warn you and say like, hey, you shouldn't be
[44:02.290 --> 44:07.470]  delegating max, like you should be delegating 90% of max or whatever you want to do.
[44:09.410 --> 44:12.750]  So with all those things we talked about with redundancy,
[44:12.750 --> 44:16.270]  I wanted to ask the question, should you be running a validator?
[44:18.950 --> 44:26.050]  And in my opinion, no, not for the validator. If centuries and things like that, where it's
[44:26.050 --> 44:32.070]  not actually signing transactions or dealing with value, that is great for scaling up and
[44:32.070 --> 44:35.850]  auto scaling. And as long as you can take some of the data considerations and things I was talking
[44:35.850 --> 44:42.570]  about earlier into that system, it's amazing for centuries. I fully recommend GCP, Amazon,
[44:42.570 --> 44:48.210]  DigitalOcean. It takes the onus off you and it gives you all of those redundancy and good,
[44:48.210 --> 44:51.950]  warm fuzzies. But for the validator, I feel like that's something that you should be controlling
[44:52.150 --> 44:57.750]  a little bit more manually doing your own internal orchestration and things like that.
[44:58.050 --> 45:02.050]  You want to be careful about where you're hosting that, who has access to it,
[45:02.050 --> 45:08.390]  what is their security posture? What is their uptime in their SLA? If their SLA is more than a
[45:08.390 --> 45:12.450]  day, that's not good for a validator because if you have a downtime event, you could be slashed.
[45:12.570 --> 45:16.470]  All things to think about when you're negotiating with your service providers.
[45:17.230 --> 45:22.330]  The last thing for best practices I wanted to talk about was the cryptocurrency standard. And this
[45:22.330 --> 45:27.690]  was more in line with what I talked about last year here at the Blockchain Village. But it's
[45:27.690 --> 45:33.210]  basically a standard that's amazing that was presented by C4 that dives into a lot of different
[45:33.210 --> 45:39.050]  security controls and aspects that one can implement in a cryptocurrency information system.
[45:39.990 --> 45:45.270]  It gives you the guidance, it gives you a control and different levels from level one to three to
[45:45.270 --> 45:50.690]  say, here's the very basic control I can put into place versus level three. Here's the ultimate
[45:50.690 --> 45:57.170]  paranoid control and kind of give you an example for that. One would be the key generation. So we
[45:57.170 --> 46:02.510]  were talking about keys earlier with the keys and which ones were both for consensus and which ones
[46:02.510 --> 46:07.050]  do transactions. And one of the things of the cryptocurrency security standard is how are those
[46:07.050 --> 46:13.210]  generated? What was the device that you generated them on? What was the entropy device? Did you do
[46:13.210 --> 46:18.010]  it on your network connected computer that you do work and Netflix and torrenting and all that stuff
[46:18.010 --> 46:23.850]  on? Or did you do it on an air-gapped machine with a secure clean room type of setup? Did you back it
[46:23.850 --> 46:29.310]  up correctly? So level one may say, I made a backup of the key, I'm good right now. It's on paper
[46:29.310 --> 46:35.810]  somewhere. But level three would say, that backup needs to be sealed, it needs to be fireproof,
[46:35.810 --> 46:40.670]  it needs to be waterproof, environmental proof, it needs to be access controlled. So as you kind
[46:40.670 --> 46:44.010]  of go up the stack here, there's different requirements and different levels of security
[46:44.010 --> 46:49.070]  and paranoia. And while you may not be able to use the cryptocurrency security standard
[46:49.070 --> 46:56.410]  to certify your information system, depending on how it works or what you're doing, it's a great
[46:56.410 --> 47:00.990]  framework to check your validator against for different levels of compliance. And it also gives
[47:00.990 --> 47:07.390]  you some really good ideas for things like data sanitization policy. So if you are spinning
[47:07.390 --> 47:12.810]  instances up and down, where are your logs going? Do you have a process for clearing all of that
[47:12.810 --> 47:17.790]  stuff correctly and securely so that nobody can run forensics on it if they wanted to profile you
[47:17.790 --> 47:22.230]  or get more information? So there's a lot of great things in here that I think definitely
[47:22.230 --> 47:26.350]  apply to that information system, even if it's not every single one of them.
[47:28.050 --> 47:34.550]  So what's next for the future? And this is really kind of exciting for me and validators and Cosmos.
[47:34.690 --> 47:41.350]  IBC is right around the corner. And I just took place in Game of Zones, which was a attack,
[47:41.350 --> 47:48.970]  defend scaling competition that the Interchain team and Zacky Mannion and Jack Sampling
[47:50.870 --> 47:55.910]  threw. So we had a whole bunch of different validators and people that were taking place
[47:55.910 --> 48:01.030]  in this. And as you can see in this graph here, these were all the participants. And they were
[48:01.030 --> 48:06.130]  connected to a main hub, a main Cosmos hub. So this is very much indicative of the wheel and
[48:06.130 --> 48:11.690]  spoke infrastructure that I think everybody's kind of used to. But this was to theoretically
[48:11.690 --> 48:16.170]  prove that all of these people could connect into the main hub and get them prepared for the next
[48:16.170 --> 48:21.730]  thing, which is the IBC protocol. And when we're talking about the IBC protocol, we're talking
[48:21.730 --> 48:27.930]  about validating on chains that we're not running necessarily. All we're doing is syncing state with
[48:27.930 --> 48:35.010]  them through light clients. So that if Alex chain, or I'm sorry, if Bob chain, for example, wanted to
[48:35.010 --> 48:40.470]  hold Alice coin, Bob and Alice's two chains can now talk together through a light client through
[48:40.470 --> 48:46.130]  IBC. So now we're breaking away from that hub and spoke model. And our validators can now choose who
[48:46.130 --> 48:50.530]  they want to talk to and where they want to validate. So you're seeing here the very beginning
[48:50.530 --> 48:55.230]  of this, where different zones are kind of breaking off into their own sovereign nations
[48:55.630 --> 48:59.570]  and saying, I want to validate on this chain, or I want to participate on this chain in addition to
[48:59.570 --> 49:04.170]  my own. And this is the magic where I was talking about earlier for validators, where when they go
[49:04.170 --> 49:08.470]  to do the rewards, they could be validating on Bitcoin, they could be validating Litecoin,
[49:08.470 --> 49:13.410]  they could be validating Ethereum and Adams, as well as Ron, Bob, Alice and Jim coin, because
[49:13.410 --> 49:18.750]  they see some value there. And when you start to open up that ecosystem and that level of connectivity,
[49:18.750 --> 49:22.630]  there's some really, really cool applications and cases and things that
[49:22.630 --> 49:26.010]  people are talking about and playing with. And it gets me really excited.
[49:27.590 --> 49:30.810]  So that's what I have for my talk. If you guys do have any questions,
[49:30.810 --> 49:34.630]  feel free to reach out to me at my consulting company, Stoner Consulting.
[49:34.990 --> 49:40.010]  I love talking about this stuff. I'd love to go into detail if anybody wants to jump more into it.
[49:40.010 --> 49:44.590]  And I also wanted to throw a special thanks out to Sean Martin, one of the people that's worked
[49:44.590 --> 49:51.370]  with me on validators and infrastructure and system. And he's really an infrastructure magician.
[49:51.370 --> 49:55.830]  So if you guys ever have infrastructure questions or scaling issues or anything like that,
[49:55.830 --> 49:58.270]  please let me know and I will get you in contact with Sean.
[49:58.730 --> 50:01.190]  I'm Ron Stoner and thank you for attending my talk.
[50:50.840 --> 50:52.320]  I don't know if he can, but I can.
[50:57.110 --> 51:00.970]  Sorry, I was looking to see if there was any questions. So I did see one in there,
[51:00.970 --> 51:06.030]  correction, yes, it is called a liveness check. That is correct. And that was in regards to the
[51:06.930 --> 51:10.630]  violations, the consensus violation and the liveness violation. So downtime.
[51:10.670 --> 51:13.550]  So that is the correct terminology for it. Thank you.
[51:20.790 --> 51:24.210]  Cool. And I don't think I see any other questions unless anyone has any.
[51:55.140 --> 51:58.060]  I would love to. Yeah. Thank you, Jay. Thank you, Nathan.
[51:58.060 --> 52:02.180]  Thank you, everybody that put on this village and promoting blockchain. I love it.
